Efficient Discriminative Convolution Using Fisher Weight Map

Convolutional neural networks (CNNs) have been studied for a long time, and recently gained increasingly more attention. Deep CNNs have especially achieved remarkably high performance on many visual recognition tasks due to their high levels of flexibility. However, since CNNs require numerous parameters to be tuned via iterative operations through layers, their computational cost is immense. Moreover, they often require a huge number of training samples and technical tricks, such as unsupervised pretraining and heuristic tuning, to successfully train the system. In this work, we present a very simple method of layer-wise convolution. We can obtain discriminative filters by using a Fisher weight map, which well separates convolved images between categories. This operation can be deterministically solved as a simple eigenvalue problem and no back propagation or hyper-parameters are needed. Because our method is layer-wise and based on a simple eigenvalue problem, it is computationally efficient. Also, it is relatively stable with a moderate amount of training samples and is capable of learning densely from high-dimensional descriptors without dropping connections between neurons, which is a common practice in conventional implementations of CNNs. We demonstrated the promising performance of our method in extensive experiments with two datasets. Our network used together with appropriate pooling and rectification techniques achieved remarkably high performance that was distinctly comparable to those that were state-of-the-art.